Hierarchies of Self-Organizing Maps for Action Recognition

We propose a hierarchical neural architecture able to recognise observed human actions. Each layer in the architecture represents increasingly complex human activity features. The first layer consists of a SOM which performs dimensionality reduction and clustering of the feature space. It represents the dynamics of the stream of posture frames in action sequences as activity trajectories over time. The second layer in the hierarchy consists of another SOM which clusters the activity trajectories of the first-layer SOM and learns to represent action prototypes. The third - and last - layer of the hierarchy consists of a neural network that learns to label action prototypes of the second-layer SOM and is independent - to certain extent - of the camera’s angle and relative distance to the actor. The experiments were carried out with encouraging results with action movies taken from the INRIA 4D repository. In terms of representational accuracy, measured as the recognition rate over the training set, the architecture exhibits 100% accuracy indicating that actions with overlapping patterns of activity can be correctly discriminated. On the other hand, the architecture exhibits 53% recognition rate when presented with the same actions interpreted and performed by a different actor. Experiments on actions captured from different view points revealed a robustness of our system to camera rotation. Indeed, recognition accuracy was comparable to the single viewpoint case. To further assess the performance of the system we have also devised a behavioral experiments in which humans were asked to recognize the same set of actions, captured from different points of view. Results form such a behavioral study let us argue that our architecture is a good candidate as cognitive model of human action recognition, as architectural results are comparable to those observed in humans

Action Recognition based on Hierarchical Self-Organizing Maps

We propose a hierarchical neural architecture able to recognise observed human actions. Each layer in the architecture represents increasingly complex human activity features. The first layer consists of a SOM which performs dimensionality reduction and clustering of the feature space. It represents the dynamics of the stream of posture frames in action sequences as activity trajectories over time. The second layer in the hierarchy consists of another SOM which clusters the activity trajectories of the first-layer SOM and thus it learns to represent action prototypes independent of how long the activity trajectories last. The third layer of the hierarchy consists of a neural network that learns to label action prototypes of the second-layer SOM and is independent - to certain extent - of the camera's angle and relative distance to the actor. The experiments were carried out with encouraging results with action movies taken from the INRIA 4D repository. The architecture correctly recognised 100% of the actions it was trained on, while it exhibited 53% recognition rate when presented with similar actions interpreted and performed by a different actor

Simulating Actions with the Associative Self-Organizing Map

Abstract. We present a system that can learn to represent actions as well as to internally simulate the likely continuation of their initial parts. The method we propose is based on the Associative Self Organizing Map (A-SOM), a variant of the Self Organizing Map. By emulating the way the human brain is thought to perform pattern recognition tasks, the A-SOM learns to associate its activity with different inputs over time, where inputs are observations of other’s actions. Once the A-SOM has learnt to recognize actions, it uses this learning to predict the continuation of an observed initial movement of an agent, in this way reading its intentions. We evaluate the system’s ability to simulate actions in an experiment with good results, and we provide a discussion about its generalization ability. The presented research is part of a bigger project aiming at endowing an agent with the ability to internally represent action patterns and to use these to recognize and simulate others behaviour

Recognizing Actions with the Associative Self-Organizing Map

When artificial agents interact and cooperate with other agents, either human or artificial, they need to recognize others\u2019 actions and infer their hidden intentions from the sole observation of their surface level movements. Indeed, action and intention understanding in humans is believed to facilitate a number of social interactions and is supported by a complex neural substrate (i.e. the mirror neuron system). Implementation of such mechanisms in artificial agents would pave the route to the development of a vast range of advanced cognitive abilities, such as social interaction, adaptation, and learning by imitation, just to name a few. We present a first step towards a fully-fledged intention recognition system by enabling an artificial agent to internally represent action patterns, and to subsequently use such representations to recognize - and possibly to predict and anticipate - behaviors performed by others. We investigate a biologically-inspired approach by adopting the formalism of Associative Self-Organizing Maps (A-SOMs), an extension of the well-known Self-Organizing Maps. The A-SOM learns to associate its activities with different inputs over time, where inputs are high-dimensional and noisy observations of others\u2019 actions. The A-SOM maps actions to sequences of activations in a dimensionally reduced topological space, where each centre of activation provides a prototypical and iconic representation of the action fragment. We present preliminary experiments of action recognition task on a publicly available database of thirteen commonly encountered actions with promising results

Simulating music with associative self-organizing maps

We present an architecture able to recognise pitches and to internally simulate likely continuations of partially heard melodies. Our architecture consists of a novel version of the Associative Self-Organizing Map (A-SOM) with generalized ancillary connections. We tested the performance of our architecture with melodies from a publicly available database containing 370 Bach chorale melodies. The results showed that the architecture could learn to represent and perfectly simulate the remaining 20% of three different interrupted melodies when using a context length of 8 centres of activity in the A-SOM. These promising and encouraging results show that our architecture offers something more than what has previously been proposed in the literature. Thanks to the inherent properties of the A-SOM, our architecture does not predict the most likely next pitch only, but rather continues to elicit activity patterns corresponding to the remaining parts of interrupted melodies by internal simulation